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LONG-TERM  GOALS 

The  long-term  goal  of  this  research  is  to  construct  the  Navy’s  next-generation  global  numerical 
weather  prediction  (NWP)  model  using  new  numerical  methods  specifically  designed  for  distributed- 
memory  and  vector  computers.  To  distinguish  it  from  the  current  Navy  global  atmospheric  model,  the 
navy  Operational  Global  Atmospheric  Prediction  System  (NOGAPS),  we  shall  refer  to  this  new  model 
as  the  Navy’s  Spectral  Element  Atmospheric  Model  (NSEAM).  To  take  full  advantage  of  distributed- 
memory  computers,  the  spherical  global  domain  of  NSEAM  is  partitioned  into  local  sub-domains,  or 
elements,  which  can  then  be  solved  independently  on  multiple  processors.  The  numerical  methods 
used  on  these  sub-domains  must  be  not  only  local  in  nature  but  also  high-order  accurate  and  highly 
efficient.  Thus,  NSEAM  is  being  constructed  so  that  it  is  as  accurate  as  the  current  spectral  model 
(NOGAPS);  is  more  efficient,  thereby  allowing  for  finer  resolution  forecasts;  and  is  geometrically 
more  flexible,  thereby  allowing  for  the  use  of  adaptive  or  telescoping  grids.  This  will  allow  for  better 
coupling  with  mesoscale  models  and  eventually  perhaps  allowing  for  a  modeling  paradigm  whereby 
the  global  and  mesoscale  models  are  either  unified  or  at  least  share  the  same  numerical  algorithms 
which  facilitates  their  maintenance. 

OBJECTIVES 

The  objective  of  this  project  is  to  construct  new  high-order  local  methods  for  the  Navy’s  next- 
generation  global  NWP  model.  The  high-order  accuracy  of  these  methods  will  ensure  that  the  new 
model  yields  the  same  accuracy  as  the  current  spherical  harmonics  model  while  the  locality  of  these 
methods  will  ensure  that  the  efficiency  of  the  model  increases  as  the  number  of  processors  increases. 

APPROACH 

To  meet  our  objectives  we  explore: 

1 .  spectral  element  (SE)  and  discontinuous  Galerkin  (DG)  methods  on  quadrilateral  and  triangular 
grids, 

2.  semi-implicit  (SI)  and  semi-Lagrangian  (SL)  time-integrators, 

3.  adding  the  NOGAPS  physics  package  to  the  NSEAM  dynamics,  and 

4.  the  nonhydrostatic  version  of  NSEAM. 

The  power  of  SE  and  DG  methods  is  that  they  are  high-order  accurate,  like  spherical  harmonics 
methods,  yet  are  completely  local  in  nature  -  meaning  that  the  equations  are  solved  independently 
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within  each  individual  element  and  processor.  Furthermore,  high-order  methods  have  minimal 
dispersion  error  which  is  an  important  property  for  capturing  fine-scale  atmospheric  phenomena  (e.g., 
tropical  cyclones,  Kelvin  and  Rossby  waves).  In  addition,  SI  and  SL  methods  offer  vast  improvements 
in  efficiency  due  to  the  longer  time  steps  that  they  permit. 

After  validation  of  the  SE,  DG,  SI,  and  SL  discretization  on  barotropic  test  cases,  the  full  3D  adiabatic 
(dry  physics)  hydrostatic  primitive  equation  model  was  closely  scrutinized.  The  SI  time-integrator  of 
NSEAM  was  ported  to  the  message-passing  interface  (MPI).  At  this  point  direct  comparisons  with  the 
current  version  of  NOGAPS  were  made.  Superior  performance  of  NSEAM  over  NOGAPS  then 
justified  the  further  effort  into  taking  the  steps  to  make  NSEAM  the  next  operational  NWP  model.  In 
order  to  accomplish  the  final  objective,  the  following  steps  must  be  accomplished: 

1 .  the  NOGAPS  physical  parameterization  package  must  be  coupled  with  the  NSEAM  dynamical 
core, 

2.  non-reflecting  boundary  conditions  need  to  be  added  to  NSEAM  in  order  to  avoid  spurious 
wave  reflections  emanating  from  the  orography,  and 

3.  development  of  the  tangent  linear  and  adjoint  models  of  NSEAM  to  be  used  for  both  data 
assimilation  and  sensitivity  studies. 

After  this  point,  the  modeling  system  can  be  said  to  be  complete.  This  year  we  have  successfully 
integrated  the  NOGAPS  physics  into  the  NSEAM  dynamics.  However,  so  far  we  have  only  tested  the 
physics  with  very  simple  initial  conditions,  such  as  the  aqua-planet  and  Rossby-Haurwitz  wave 
experiments  -  both  of  which  do  not  use  real  data  but  rather  simplified  initial  conditions.  In  order  to  be 
able  to  include  real  data,  various  input  routines  must  be  developed  such  as  the  climatological  data 
(ground  wetness,  albedo,  etc.)  which  we  have  developed  for  a  previous  version  of  NSEAM.  The  final 
step  will  include  the  construction  of  absorbing  boundary  conditions,  as  well  as  tangent  linear  and 
adjoint  models. 

WORK  COMPLETED 

NSEAM  Numerics.  The  main  accomplishments  in  numerics  this  year  involved  the  development  of  a 
new  class  of  diagonal  mass  matrix  triangular  spectral  element  method  (see  Ref.[l]).  Before  this  work, 
SE  methods  were  only  used  on  quadrilaterals  and  if  they  were  used  on  triangles  they  did  not  possess  a 
diagonal  mass  matrix  which  is  the  key  to  the  efficiency  of  the  SE  method.  With  this  new  technology, 
we  have  built  shallow  water  models  on  the  sphere  and  we  now  propose  to  include  this  triangular  SE 
method  within  the  existing  version  of  NSEAM  with  little  modification  to  the  model. 

Physical  Parameterization,  The  majority  of  the  time  this  year  was  spent  on  coupling  the  NSEAM 
dynamical  core  with  the  current  NOGAPS  physical  parameterization  routines  (called  DIABAT). 
During  PY2006  two  separate  approaches  were  used  to  couple  the  NSEAM  dynamics  with  the 
NOGAPS  physics.  Eirst,  a  single-column  model  (SCM)  version  of  the  NOGAPS  physics  was  placed 
inside  of  NSEAM.  This  was  a  proof-of-concept  approach  to  show  that  the  NOGAPS  physics  would 
work  “out  of  the  box”  within  NSEAM.  With  this  approach,  we  performed  the  aqua-planet  experiment 
and  our  results  compared  quite  favorably  with  those  found  in  the  literature.  However,  this  approach 
proved  too  costly  to  use  in  an  operational  setting  and  a  second  approach  was  devised. 
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In  the  current  approach,  the  NOGAPS  physics  routine  (DIABAT)  was  removed  from  NOGAPS, 
minimal  modifications  were  made,  and  then  installed  directly  into  NSEAM.  The  results  so  far  have 
been  quite  impressive.  The  simple  experiments  run  with  this  full  model  have  shown  to  yield 
physically  realistic  results  and  the  cost  between  the  physics  and  dynamics  is  roughly  one-to-one,  which 
is  what  one  would  hope.  We  are  now  ready  to  tackle  the  aqua-planet  and  Madden-Julian  Oscillation 
(MJO)  experiments  with  this  new  model.  The  main  advantage  of  this  approach  is  that  any  changes 
made  to  the  NOGAPS  physics  can  be  readily  included  into  NSEAM.  We  now  propose  to  compare  the 
efficiency  of  NSEAM  with  NOGAPS  physics  to  the  NOGAPS  system.  Since  the  cost  between  physics 
and  dynamics  is  roughly  one-to-one  this  then  means  that  NSEAM  will  continue  to  scale  far  better  than 
NOGAPS,  as  the  dry  dynamics  tests  indicated  previously. 

An  important  issue  raised  when  we  coupled  the  NOGAPS  physics  with  the  NSEAM  dynamics  is  the 
creation  of  noise  in  the  precipitation  fields  along  the  element  boundaries  of  the  spectral  elements.  This 
is  not  an  unexpected  phenomenon  but,  nonetheless,  it  is  a  situation  that  must  be  rectified.  In  order  to 
control  this  “spectral  rain”  we  have  developed  sophisticated  diffusion  operators  for  the  spectral 
element  method;  this  is  in  fact  a  new  area  of  research  within  the  spectral  element  community  and  we 
have  made  great  advances  in  creating  these  required  high-order  diffusion  operators.  If  we  use  2"^* 
order  diffusion,  then  the  fields  are  too  damped  with  little  regard  for  the  low  frequency  (long)  waves. 
However,  if  we  apply  4*^  order  or  higher  hyper-viscosity,  then  we  are  guaranteed  to  keep  the  low 
frequency  waves  untouched.  Regardless  of  which  diffusion  operator  we  use,  we  can  now  eliminate  the 
effects  of  the  so-called  “spectral  rain”.  A  paper  on  this  topic  is  in  preparation. 

RESULTS 

NSEAM  Numerics.  At  the  present  time,  all  spectral  element  models  use  quadrilateral  elements  for 
partitioning  the  grid.  Quadrilaterals  have  worked  very  well  in  the  past  but  they  are  not  as  geometrically 
flexible  as  triangles.  The  issue  with  triangles  is  that  no  one  had  yet  devised  a  high-order  (spectral 
element)  triangular  element  approach  that  has  the  same  high  efficiency  of  quadrilateral-based  spectral 
elements.  We  have  developed  a  “true”  triangular  spectral  element  method.  The  advantages  of  using 
triangular  elements  is  that  we  can  now  discretize  the  domain  emphasizing  specific  regions  of  interest. 


Figure  1:  New  class  of  adaptive  unstructured  grids  possible  wit 
the  triangular  spectral  element  method. 

In  Eigure  1  we  show  an  adaptive  unstructured  grid  where  the  grid  has  been  refined  such  that  high 
resolution  is  obtained  on  the  land  masses  and  lower  resolution  along  the  oceans.  However,  we  could 
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also  reverse  the  situation  and  obtain  high  resolution  near  specific  regions  of  the  oceans,  for  example,  in 
order  to  track  tropical  cyclones  more  accurately.  In  Figure  2,  we  show  the  convergence  rate  of  the  new 
triangular  spectral  element  method.  For  a  method  to  be  considered  spectral,  the  method  must  converge 
at  a  rate  of  (N+1).  In  other  words,  for  a  fixed  polynomial  order,  N,  the  error  must  decrease  as  a  power 
of  (N+1)  when  we  add  more  elements.  This  is  the  rate  of  convergence  that  is  reported  in  Figure  2. 

Note  that  for  all  orders  of  N  we  can  indeed  see  that  the  order  is  between  N  and  N+1 .  We  expect  to  use 
the  N=6  which  is  a  sixth  order  polynomial  that  achieves  roughly  seventh  order  accuracy.  We  chose 
this  value  because  in  Ref.  [1]  we  show  that  this  value  yields  the  highest  accuracy  for  the  lowest 
computational  cost. 


Figure  2:  The  convergence  rate  of  the  new  triangular  spectral  element  method.  This  shows  that  the 
method  does  indeed  converge  at  an  exponential  (spectral)  rate. 


Physical  Parameterization. 


In  order  to  validate  whether  the  NSEAM  dynamics  run  with  NOGAPS  physics  is  working  properly,  we 
ran  the  Rossby-Haurwitz  wave  number  4.  For  pure  (dry)  dynamics,  NSEAM  maintains  the  wave  4 
structure  indefinitely  (up  to  at  least  30  days).  However,  when  the  NOGAPS  physics  is  added,  the 
dynamics  of  the  wave  changes  quite  drastically. 
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Figure  3:  300mb  wind. 
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In  fact,  we  can  see  that  at  300  mb  (Figure  3)  the  wind  field  develops  a  very  strong  jet.  This  simple  test 
confirms  that  the  dynamics  and  physics  are  working  well  together  to  produce  physically  realistic 
results. 
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a)  Surface  Temperature  Forcing 


b)  Convective  Precipitation 


Figure  4:  The  surface  temperature  and  resulting  convective  precipitation  for  NSEAM  dynamics 
with  NOGAPS  physics  for  the  Aqua-Planet  Experiment.  The  model  was  run  successfully  for  a  3 
year  simulation  but  here  we  only  show  results  after  6  months. 


In  a  second  test,  we  run  the  aqua-planet  experiment.  This  experiment  uses  a  specified  sea-surface 
temperature  as  the  forcing  mechanism  which  then  drives  the  planetary  climate.  This  sea-surface 
temperature  is  shown  in  Figure  4a.  This  forcing  eventually  causes  precipitation  to  occur  near  the 
equatorial  region.  The  convective  precipitation  averaged  over  days  120  to  180  are  shown  in  Figure  4b. 
These  results,  along  with  other  detailed  results  such  as  Hough-Moeller  and  Wheeler-Kiladis  diagrams 
(see  Ref.  [8]),  show  the  generation  of  Rossby  and  Kelvin  waves  which  suggests  that  the  dynamics  and 
physics  are  interacting  properly. 

IMPACT 

NOGAPS  is  run  operationally  by  FNMOC  and  is  the  heart  of  the  Navy’s  operational  support  to  nearly 
all  DOD  users  worldwide.  This  work  targets  the  next-generation  of  this  system  for  massively  parallel 
computer  architectures.  NSEAM  has  been  designed  specifically  for  these  types  of  computer 
architectures  while  yielding  the  same  high-order  accuracy  as  NOGAPS. 

TRANSITIONS 

Improved  algorithms  for  model  processes  will  be  transitioned  to  6.4  (PE  0603207N)  as  they  are  ready, 
and  will  ultimately  be  transitioned  to  ENMOC  with  future  NOGAPS  upgrades. 

RELATED  PROJECTS 

Some  of  the  technology  developed  for  this  project  will  be  used  immediately  to  improve  the  current 
spectral  transform  formulation  of  NOGAPS  in  other  NRL  projects. 
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